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METHOD FOR SIMULATING FILM GRAIN BY MOS AICING PRE-COMPUTED 

SAMPLES 

CROSS-REFERENCE TO RELATED APPLICATIONS 

This application claims priority under 35 U.S.C. 1 19(e) to U.S. Provisional Patent 
Application Serial No. 60/505,146, filed on September 23, 2003, the teachings of which are 

» . 

incorporated herein." 
10 TECHNICAL FIELD 

This invention relates to a technique of for simulating film grain in an image. 
BACKGROUND ART 

15 

Prior publications authored by the present inventors and assigned to the present 
assignee proposed simulating film grain in a decoded video stream by first filtering grain out 
the image before compression. The video stream undergoes compression and subsequent 
transmission to a decoder for receipt along with a message containing information about the 
20 film grain that was present in the stream prior to compression. Upon receipt of the 

compressed video stream and grain-containing message, the decoder decodes the compressed 
stream and then restores the original grainy appearance of the image by simulating the film 
grain based on the content of the grain information message. The film grain message can take 
the form of a Supplemental Enhancement Information (SEI) message accompanying the 

25 coded video stream. 

Simulating film grain in this manner provides large bit-rate savings for high quality 
applications where the film grain preservation becomes important. However, this method of 
simulating film grain increases the decoder complexity since the.decoder must reproduce and 
blend the film grain with the decoded video stream as specified by the transmitted film grain 
30 information message. 

Thus, a need exists for a technique for simulating film grain that overcomes the 

disadvantages of the prior art. 



WO 2005/034518 



PCT7US2004/029410 



-2- 

BRIEF SUMMARY OF THE INVENTION 

Briefly, the present invention provides a method for simulating film grain in an input 
image block. The method commences by first computing an average value of at least one . 
image parameter for the input image block. Thereafter, a film grain block is selected from at 
least one previously established pool of film grain blocks whose image parameter most 
closely matches the image parameter of the input image block. The selected block is then 
blended with the input image block. 

Selecting a film grain block from the at least one pool of pre-established film grain 
blocks for combination with the input image reduces the complexity associated with 
simulating the film grain at a decoder as was done by prior methods. Further, selecting a film 
grain block from the at least one pool of pre-established film grain blocks reduces artifacts 
that otherwise might arise when transitioning between independently generated film grain 
blocks. 

BRIEF DESCRIPTION OF THE DRAWINGS 

FIGURE 1 depicts the steps of a prior art method for film grain simulation by 
mosaicing independently generated film grain samples; 

FIGURE 2 depicts the steps of a prior art method for generating the film grain blocks 
for the method of FIG. 1 ; 

FIGURE 3 depicts a part of a prior art decoder for generating a film-grain containing 

output image; 

FIGURE 4 depicts the steps of a method in accordance with an illustrative 
embodiment of the present principles for simulating film grain by mosaicing pre-computed 
blocks of film grain; 

FIGURE 5 depicts the steps of a method in accordance with an illustrative 
embodiment of the present principles for pre-establishing the film grain blocks used in the 

method of FIG. 4; and 

FIGURE 6 depicts the steps of a method in accordance with another illustrative 
embodiment of the present principles for simulating film grain by mosaicing film grain blocks 
obtained from input image characteristics. 
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DET AILED DESCRIPTION 

In accordance with the present principles, simulation, of film grain in an image from 
.occurs by mosaicing pre-established individual film blocks. The term "mosaicing" of the 
5 individual film bloclcs implies the building of a composite image by blending individual film 
grain blocks that are smaller in size. To appreciate the advantages of film grain simulation by . 
mosaicing jire-established film grain samples, a description of the prior art film grain 
simulation process will prove helpful. 

FIGURE 1 depicts the steps of a prior art method for simulating film grain on a block- 
10 by-block basis. The method of FIG. 1 commences by first initializing system variables during 
step 10. During step 12, a successive one of a set of blocks of N x M pixels (where N and M 
are each integers greater than zero) is read from in an input image 13. Reading of the blocks 
during step 12 occurs in raster-scan order. Prior to step 12, the input image 13 typically 
undergoes filtering to remove or at least attenuate the film grain followed by decomposition to 
15 yield non-overlapping blocks of N x M pixels. Initial compression of the image can itself 

serve to remove or attenuate film grain. In some instances the input image will not have any 
initial film grain, either because the image was digitally captured and/or, computer generated. 
• Under such circumstances, adding grain where none existed will enhance the image. . 

Following step 12, the numeric average of the image intensity, is computed during step 
20 14 for the just-read block. Next, film grain parameter selection occurs during step 16. The 
selection is made from a set of film grain parameters provided in a film grain message 17 
accompanying the input image in accordance with the average intensity value computed 
during step 14. Typically the film grain message 17 takes the form of a Supplemental 
Enhancement (SEI) Message, and for that reason, the term "SEI message" will appear 
25 hereinafter when referring to the message containing film grain information. Because film 

grain characteristics can vary depending on the intensity level, the selection carried out during 
step 16 can yield different film grain parameters depending on the .average intensity measured 
on the extracted image block. 

Using the film grain parameters selected during step 16, an N x M pixel block of film 
30 grain samples is generated during step 18 for use in creating a film grain image 19. Each of 
the film grain blocks generated through this process undergoes storage and compositing to 
yield a film grain image that maps to the size of the original input image. This composition 
process constitutes mosaicing as discussed earlier. Following step 18,. a check occurs during 
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step 20 whether the block selected during step 12 constitutes the last block of the input image 
1 1. If additional blocks remain, program execution branches to step 12 and those following it. 
Otherwise, step. 22 occurs and the process ends 

FIGURE 2 depicts the individual steps that collectively comprise the step 18 of FIG. 1 
5 for generating block of N x M film grain samples. The method of FIG. 2 commences upon 
execution of step 200 during which initialization of system variables occurs. Thereafter, 
generation of a block of N x M random values occurs during step 202 using one or more 
parameters obtained from the SEI message 17. The block generated during step 204 
undergoes a Discrete Cosine Transformation (DCT) during step 204 to obtain a corresponding 
10 set of N x M frequency coefficients. Other transforms, besides DCT could also be used. The 
coefficients computed during step 204 undergo frequency filtering during step 206, with the 
filtering coefficients computed in accordance with one or more of the parameters contained in 
the SEI message 17. The frequency filtering controls the size of the simulated grain. The 
frequency-filtered coefficients undergo an Inverse Discrete Cosine Transform (IDCT) during 
15 step 208 to yield the film grain image 19. This process requires specification of the noise 
deviation as well as the high and low cut frequencies that control the filtering process in the 
frequency domain. For video coding applications, the SEI message 17 typically conveys such 
information. Following step 208, the process ends. 

FIGURE 3 depicts a block schematic diagram of a portion of a decoder 300 that 
20 accomplishes film grain blending with the original image. A summing block 310 within the 
decoder 300 receives a decoded input image 13 at a first input. The film grain image 19 of 
FIG. 2 undergoes deblocking at a de-blocking filter 314 prior to receipt at the second input of 
the summing block 310. The summing block 310 sums the input image 13 and the film grain 
image 19 to yield an output image 316 that contains film grain. Since the individual blocks of 
25 film grain have been generated independently, artifacts can be perceived at the transitions 
between blocks. Thus, the de-blocking filter 314 becomes important to reduce the visual 
artifacts resulting from the film grain mosaicing. 

As thus described, the prior art method for film grain simulation requires the 
generation of film grain for blending with each processed block of pixels of the input image. 
30 In contrast, the film grain simulation in accordance with the present principles achieves 

greater efficiency by limiting the film grain generation by way of a pool creation process in 
which a limited number of film grain blocks are computed. This approach strongly reduces 
the computational complexity of the film grain simulation process. 
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FIGURE 4 depicts the steps of a method in accordance with a first embodiment of the 
present principles for film grain simulation. As discussed in greater detail hereinafter, the 
film grain simulation method of FIG. 5 advantageously avoids film grain generation on the fly 
by selecting individual pre-computed film grain blocks from a pool of such blocks. The 
method of FIG. 4 commences by first initializing system variables during step 400. Next a 
successive block of N x M pixels is read during step 402 from the input image 13. Reading of 
the blocks during step 402 occurs in raster-scan order. As discussed above, the input image 
13 typically will have had all of its grain removed or attenuated and will have been 
decomposed into non-overlapping blocks of N x M pixels (where N and M are integers each 
greater than zero). Initial compression of the image can itself serve to remove or attenuate 
film grain. In some instances the input image will not have any initial film grain, either 
because the image was digitally captured and/or, computer generated. Under such 
circumstances, adding grain where none existed will enhance the image. 

Following step 402, the average value of an image parameter, typically, the average 
intensity value is computed during step 404 for the block just read during step 402. Next, a 
film grain block is selected during step 406 from at least one pool of pre-established film 
grain blocks pre-computed in the manner described hereinafter with respect to FIG. 5. Rather 
than rely on a single pool of film grain blocks, multiple pools could exist from which selection 
could occur dependent on the average image intensity or dependent on one or more other 
image parameters. When more than one block is available for the same intensity level and 
color component, a selecting criterion should be specified. For example, pseudo-random 
selection of blocks from the pool could occur to avoid the creation of patterns when a reduced 
number of blocks are available. Use can also be made of transformed copies of the set of 
available film grain blocks. 

Following block selection during step 406, the selected block typically undergoes a 
transform during step 410 to yield the film grain image block 19 for blending with the film 
grain image.. Following step 410, a check occurs to determine whether additional image 
blocks remain during step 412. If additional image blocks remain, program execution 
branches to step 402 and those following it. Otherwise, program execution ends during step 
414. 

FIGURE 5 illustrates the steps of a method in accordance with the present principles 
for generating the pre-established film grain blocks 408 of FIG. 4. The method of FIG. 5 
commences by first performing initialization during step 500 to reset all system variables. 
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Next, selection of a set of film grain parameters from the SEI message 17 occurs during step 
502. Thereafter, a set of K blocks of film grain samples (where K is an integer greater than 
zero) is generated for each set of parameters in the SEI message 17 during step 504. Different 
sets of parameters will specify different color components and different intensity levels. A 
particular implementation with K=l results in the lowest computational requirements, 
generating only one block of film grain per set of parameters. However, to avoid the creation, 
of patterns, it is preferable to use of a larger number of blocks or even a different number of 
blocks for different intensity levels. 

Following step 504, a check occurs during step 506 to determine whether additional . 
parameter sets remain. If additional sets remain, program execution branches to step 502 and 
those following it. Otherwise, program execution ends during step 508. 

The process described with respect to FIG. 5 for creating a pool of pre-established film 
grain blocks remains completely decoupled from the film grain blending that occurs in the 
decoder 10 as described in FIG. 3. Despite the computational advantage of anticipating film 
grain generation, the film grain simulation method of FIG. 4 incurs the inconvenience of 
remaining unaware whether the input images will make use of all the blocks stored in the 
pool. 

FIG. 6 illustrates an alternative embodiment of a film grain simulation method 
according to the present principles that makes use of the characteristics of input images. As 
will become better understood hereinafter, the film grain method of FIG. 6 does not create a 
pool of film grain blocks a priori for all the possible intensity levels present in the SEI 
message 17. Instead, the method of FIG. 6 undertakes creation of a limited pool of film grain 
blocks and undertakes to update of the pool of blocks depending on the input image 
characteristics. 

The method of FIG. 6 commences by first performing initialization during step 600 to 
reset all system variables. Step 602 follows step 600 during which selection of a block of N x 
M pixels from the input image 13 occurs. Thereafter, the average of at least one image 
parameter (e.g., the average of the image intensity) of the previously selected image block is 
computed during step 604. During step 606 a check occurs for the availability of a film grain 
block whose average intensity most closely matches the average intensity of the selected 
image block. If no such block exists, then program execution branches to step 608, a set of 
parameters are selected from the SEI message 17. Next, an N x M film grain block is 
generated from the selected parameters during step 610. The block generated during.step 610 
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now enters the pool of pre-computed film grain blocks 408. Step 612 follows step 610. Step 
612 also follows step 606 upon a determination that a film grain block in the pool has an 
average intensity mostly closely matching that of the selected image block. During step 612, 
selection of the most closely matching film grain block occurs from the pool 408 of film grain 
5 blocks. Next, the selected block typically undergoes a transform during step 614 for 

mosaicing with the previously selected blocks to yield the film grain image 19. In this way, 
the transform block-is blended with the film grain image. A determination occurs during step 
616 whether the block selected during step 602 constitutes the last block. If not, program 
execution branches to step 602. Otherwise, program execution ends during step 618. 
10 The above-described method can estimate if blocks are not available not only when the 

pool 408 is empty, but also if the existing blocks have been recently used for blending. Other 
criteria for managing the pool creation and update process could also be envisaged. 
The method of FIG. 6 could progressively create and/or update the pool 408 of pre-established 
film grain blocks depending on the characteristics of the input images. For example, the pool 
1 5 . 408 could be organized to arrange a larger number of blocks within the most -prevalent 
intensity levels to avoid the creation of visual patterns. According to this strategy, the 
distribution of film grain blocks in the pool 408 would progressively match variations in the 
image characteristics related to darkening or lightening. 

The foregoing describes a technique for simulating film grain in an image. 



